Greedy Minimization of Weakly Supermodular Set Functions

This paper defines weak-α-supermodularity for set functions. It shows that minimizing such functions under cardinality constrains is a common task in machine learning and data mining. Moreover, any problem whose objective function exhibits this property benefits from a greedy extension phase. Explicitly, let S∗ be the optimal set of cardinality k that minimizes f and let S0 be an initial solution such that f(S0) ≤ ρf(S∗). Then, a greedy extension S ⊃ S0 of size |S| ≤ |S0|+ dαk ln(ρ/ε)e yields f(S) ≤ (1 + ε)f(S∗). Example usages of this framework give streamlined proofs and new bi-criteria results for k-means, sparse regression, column subset selection, and sparse convex function minimization. Sparse regression and column subset selection are special cases of a new, more general, sparse multiple linear regression problem that is of independent interest. This paper also corrects a brittleness of the proof of Natarajan for the properties of the greedy algorithm for sparse regression. 1998 ACM Subject Classification G.1.3 Numerical Linear Algebra, G.1.6 Optimization, G.4 Mathematical Software